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Field of th e Invention 

The present invention generally relates to a video recording method, a video display method 
and video recording apparatus, a video control program on a carrier and a video display 
5 control program for mteractive viewing. 

Background op the invention 

Conventional video is recorded and displayed in a format exhibiting a ratio of 4 to 3 as 
related to horizontal and vertical elongation. The resolution in a video image is determined 
10 by the number of picture elements (pixels), which in a conventional video is in the range of 
700 - 800 in the horizontal direction and 500 - 600 in the vertical direction. 

For the purpose of this application the word video should also be understood as a sequence 
of digital images. Wide image should be understood as a form of a panoramic image. 

15 

Prior art 

Wide panorama images composed of several individual images are know within the 
neighboring arts. 

20 Summary op the invention 

The present invention concerns a method of combining a number of simultaneous video 
sequences generated by several cameras. It also concerns an apparatus for recording of such 
a video sequence, as well as control program for creating the composed video sequences 
and control program for interactive viewing of such sequences as well as for display of the 
25 same. The invention also concerns a video player for watching the recorded video 
sequences. 

A known problem when watching field sports on e.g. television is that in order to get a 
image of the whole field and thus a possible overview the camera has to be set far away 
30 from the field and the resulting image will by necessity be very small in details. By zooming 
in on different parts of the field the camera may of course catch details but the "whole 
image" will be lost. Anyone having watched a soccer game on television will know the 
problem existing. 



According to the invention the wide image video is created digitally by combining image 
information from several cameras which together cover a wide field of vision. The wide 
image video may be thought of as an image from a virtual camera with an extremely wide 
field of vision. Every point of the scene being recorded will be recorded by at least one 
5 camera. Every point in the image from each camera corresponds to an unambiguously 
corresponding point in the synthetic wide image. By transferring the intensity of this last 
point to the synthetic wide image the wide image is formed. By intensity according to this 
invention is included the combination of colors which build up every point in the image as 
well as the ratio of each such color. 

In order to be able to transfer the intensity from each point to the synthetic wide image the 
relation between die respective coordinates for the pixels in the individual cameras and in 
the synthetic wide image must be ascertained. This relation may be described according to 
the invention mathematically by a projective transformation. The transformation may be 
15 determined through observations of the coordinates for a number of points in the scene 
which is depicted by more than one camera. 

According to the invention this step is a crucial step of the process. The cameras used are 
stationary and the relation between the images presented by each camera is determined by 
20 identifying corresponding points in overlapping parts of the images. In the case of a football 
field or any game sport where lines are indicated in the field these lines may be used for this 
purpose. 

The wide image video may be displayed on and viewed on e.g. conventional computer 
25 screens or the like. The onlooker may decide to view either die whole image to focus on 

parts of the same, by using a special program according to the invention. As is know within 
the art the computer may be connected to a projector to project the video image on a larger 
area, Le. a film screen, a smart board, or the like. The wide image video may also be cut 
into such sizes as to fit in a television screen or the side image video may also be cut into 
30 parts, each part televised over different television channels and then when received be 
projected on a screen or the like for showing all of the wide video image, or video 
sequence. 
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Viewing of the video sequence on a film screen may be performed using several projectors 
in order to preserve the high quality and the resolution. The wide image sequence is for this 
purpose split into a number of part sequences of conventional video size. These part 
sequences are thereafter displayed and projected side by side in order to form a wide 
5 projected image. 

It is thus an object of the present invention to generate a considerably wider image which 
covers a considerably wider field of vision than a conventional video image. 

10 It is also an object of the present invention to generate video sequences having a desired 
format and a desired resolution by combining sequences recorded simultaneously using 
several cameras. 

It is also an object of the present invention to generate a video sequence where the format 
1 5 exhibits a ratio of e.g. 2 to I as related to horizontal and vertical elongation with 
approximately 2500 pixels in the horizontal elongation. 

It is also an object of the present invention to provide means for choosing a specified area to 
. be moved over the wide image and to possibly enlarge the chosen area (zooming in on an 
20 interesting part of the screen). This feature may also be used in case a display is used which 

is not wide enough to hold all of the wide image. 

The present invention therefore provides a method for generating a wide image video 
sequence, said method comprising the steps of : a generating a set of calibration parameters 
25 related to a device having at least two video cameras which are arranged in a predetermined 
relationship to each olher, said parameters being unique for the at least two cameras and their 
current location as related to the object being recorded; b. recording synchronously video 
sequences using each of said at least two video cameras, and c. generating a wide image video 
sequence from each of said synchronously recorded video sequences. 

; 30 

The method preferably provides the storing of the synchronously recorded video sequences in 
: a memory means. The method also provides for the synchronously recorded video sequences 

: being concurrently used for generating the wide image video sequence. The method also 
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provides for the wide image video sequence being transmitted live. The method also provides 
for the wide image video sequence being stored on a memory means. 

f 

The invention also provides for a method for generation of calibration parameters comprising 
foe following steps: a. Start of calibration process; b. Synchronizing foe sequences from 
each camera, which means that at least a video sequence has to be recorded by all cameras; 
c. Computing inter-image projective transformations; d. Use foe transformations to refer 
each image to a common reference frame; e. Choose a real or virtual reference view such 
that certain lines on the pitch and/or stadium are essentially horizontal and parallel in the 
wide image; f . Select a rectangular region of interest within foe wide image. This region 
contains foe entire piteh and as much of foe stadium as is required or visible; and g. Record 
all computed values resulting from foe calibration process to be used as foe calibration 
parameters. In foe description and foe claims foe a pitch or a stadium is referred to as an 
example only. However this could be any wide view which one wants to cover over a wide 
IS viewing angle. 

The invention also provides for foe steps of findingfoe lens distortion parameter(s) for each 
camera, and correcting radial distortion in each image produced are comprised, and further 
provides for a step in which selection of non-linear distortion parameters to reduce 
20 perspective distortion of the wide image is comprised. 

The invention further provides for step b of foe method being performed manually by 
identification of corresponding features in concurrent video images and foe coordinates for 
these corresponding features are input to a computer means. Step b may also be performed 
25 automaticaUybyanalgori^ 

images and foe coordinates for these corresponding features are input to a computer means. 

The invention also provides for a method of recording or sending live a side video sequence 
which comprises foe following steps: a. Apply the computed and registered calibration 
parameters. For each pixel in foe wide image, compute and store parameters describing 

1 . Which pixels from which image(s) contributes to this pixel in foe wide image. 

2. How much these pixels each contribute to foe wide image, b. Repeat until foe end of the 
sequence is reached, c. Obtain one new image from each camera; d. If required, update foe 
parameters needed to transform intensities (colors/brightness) in one or more cameras to 
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eliminate visible seams; e. If necessary, adjust the intensities (colours/brightness) in the 
images from one or more cameras; f. Create the current seamless, wide image from the 
current images from each camera; g. Output the wide image to a display or to a memory 
means; and h. End of sequence. Return to step b until end of generation of the wide image 
5 video sequence* 

The invention also provides for die new images from each camera are read from live 
sources, each such source comprising a video camera or that the new images from each 
video camera are read from a memory means. 

10 

Hie present invention further provides in a device having a processor means, which 
executes instructions stored in a least one memory means the above described features. 

The present invention further provides a computer readable memory means storing a 
IS program which provides the above described features. 

The present invention is preferably realized in video recording apparatus comprising: 

a microprocessor, a memory means for storing program for generating a set of calibration 

parameters related to a device having at least two video cameras which are arranged in a 
20 predetermined relationship to each other, said parameters being unique for the at least two 

cameras and their current location as related to the object being recorded; 

said memory means also storing program for recording of wide image video sequences; 

read and write memory means for storing data relating to recorded video sequences from at 

least two video cameras; 
25 input means for input of manual input of parameters, input of recorded video sequences, 

output means for output of a wide image video sequence. 

[ Brief Description of the drawings 

; 30 In order to explain the objects, advantages and features of the present invention, reference is 
. made below the figures of the drawings, wherein: 

• '■»...,'. 

: Fig. 1 shows a schematic view of a set up of four cameras according to the invention. 

Fig. 2 shows a schematic view of a setup of six cameras according to the invention. 



Fig. 3a illustrates the cameras and overlapping recorded areas in the embodiment 
according to Fig. 1. 

Fig.3b illustrates the recorded areas of four cameras according to the invention as 
projected on the wide image. 
5 Fig. 4 shows a vertical view of two of the cameras according to the embodiment 
according to Fig. 2. 

Fig. 5 illustrates schematically the initiation part of the recording of a wide image digital 

video sequence and the recording according to the invention. 
Fig. 6a iUustratesthe TOordinatetransfo 
10 Fig. 6b illustrates the weighted image value. 

Fig. 6c illustrates how to provide a seamless transition from image 1 to image 2. 
Fig. 6d illustrates the projective transformations between the cameras. 
Fig. 6e illustrates the result of the projective transformations between the cameras. 
Fig. 7a shows the original wide image produced 
15 b shows a transformed image using small values for G^andcCy 

c shows a transformed image using medium values for Ox and oty 
d shows a transformed image using large values for ctx and Oy 
e shows a transformed image using very large values for ctx and oty 
Fig. 8 shows a flow-sheet describing a process according to the invention of generating a 
20 video sequence. 

Fig. 9 shows a flow-sheet describing an example of calibration of the cameras according 
to the invention. 

. Fig. 10 shows a flow-sheet describing an example of recording of a video sequence 
according to the invention. 
25 Fig. 1 1 shows a data processing device for performing the method according to the 
invention. 

Fig. 12 

a - b shows the selection of a part of the wide image for zooming in. 

30 Detailed description Op the Preferred Embodiments op the invention 

In a first embodiment of the apparatus of the invention the camera set up is described in 
connection with Fig. 1. The figure illustrates a combination of four (4) video cameras 101, 
102, 103, and 104. These cameras are preferably attached to a rectangular plate (not shown) 
having a horizontally aligned slit The plate is mounted on a camera tripod (not shown), and 

35 the cameras are directed towards the scene 110, being a football field, such that they cover a 



wide field of vision approximately 120 - 160 degrees. In recording events which occur over 
big plane areas (fields) the cameras are so adjusted as to depict an area each of approximately 
die same size. In this manner the quality of the wide image will be essentially uniform. 
Indicated is also the areas 111, 112, 113, and 114 covered by respective cameras. The views 
fiom the cameras pairwise overlap such that the area 1 1 1 , and 1 1 2 has a common area 1 1 5, 
the area 1 12, and 1 13 has a common area 1 16, and the area 1 13, and 1 14 has a common area 
1 17. Indicated are also the limits of the field of vision for each camera, 121, 122, 123, and 

124. "■ • : 

In Fig. 2 a corresponding set up is shown the field indicated as 210. The cameras 201, 202, 
and 203 cover the areas 221, 222, and 223, closest to the cameras and the cameras 204, 205, 
and 206 cover the areas 224, 225, and 226. 

In Fig 3a the overlap areas 315, 326, and 317 are indicated as well as the field 310, and the 
cameras 301, 302, 303, and 304 

In Fig. 3b is shown the corresponding fields on a synthesized wide image shown. The wide 
image thus will have its limitations in the vertical direction determined by the height if the 
middle cameras and in the horizontal direction by the left edge of the camera to the left and 
the right edge of the image of the camera to the right Thus the different cameras contribute to 
different parts of the wide image and also mere can be seen the there are some regions in the 
wide image covered by more man one camera. 

In Fig. 4 is schematically shown a camera set up in a vertical view. Two cameras 401 and 402 
each coveringthe areas 411 and 412, respectively. The indicated placement of the cameras in 
in the vertical position as related to the closest limit of the field may be dj approximately 20 
meters and dj approximately 20 meters. 

In Fig. 5 the pro cess of initiating the recording of a digital video sequence using the wide 
image concept is shown. In the figure can be seen two cameras 501, and 502. (This is no 
limitation only illustrating that use of more than one camera is according to the invention.) 
Also is seen memory means 503 and 504 and a personal computer 505 with key-board 506 
and a mouse 507. 
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Also shown is a program 508 residing in the non-volatile memory of the computer the main 
procedures comprised in the program for obtaining the wide image video sequence 
indicated. . 

S The procedures comprised are as follows: 

Find common points or lines in overlapping parts of the images as seen by the "n" cameras. 
Manually registering, through input of parameters, such points or lines. (This step could be 
performed also by a control program) 

Set-up process completed Start recording of digital video sequences. 
1 0 Compose the wide image from the individual images recorded. 

The composed wide video image may be stored on a volatile or non-volatile memory or it may 
be watched live. 

The recorded contents of the digital video cassettes are then transferred from the cameras 
15 involved via e.g. a fire-wire connection to a write and read memory in e.g. a personal 

computer or the like. It is of course clear that the manner of storing and treating the images to 
be computed into the wide image is not crucial to the invention and there could be several 
ways in which to perform this. 

20 In order to be able to generate a wide image sequence for the individual sequences, the 
sequences have to be projected on a common image plane. One of the image planes of the 
cameras may be chosen (the reference camera). The projection on the common image plane is 
accomplished using a co-ordinate transformation specific for each camera and each set up of 
the cameras. 

25 / 

This co-ordinate transformation is determined by noting the co-ordinates for a number of 
points which are in view in die current camera and also in the image plane of the reference 
camera simultaneously. The thus calculated co-ordinate transformation is thereafter applied to 
pixels from the current camera, which procedure gradually builds the wide image in the 
30 image plane of the reference camera. 

This transformation is illustrated in Fig. 6a. If OCX) denotes the co-ordinates of the pixel 
in a current camera and (X>Y) the co-ordinates of the corresponding pixel in the reference 



camera the following relation between these co-ordinates may be described using a 
projective transformation: 



aX + bY + c gX + hY + i 

5 X'= — Y' = — d),(II) 

dX + eY + f dX + eY + f 

The parameters a, b, c, d, e, f , , g, h, and i are determined by noting the co-ordinates for a 
number of chosen points which are can be seen in both the current camera and in the 
10 reference camera* 

When using lines instead of points the equations will differ slightly but principally they are 
the same. 

The parameters {a,b,c,d,e,f,g,h,i} can also he found linearly from pairs of corresponding 
1 5 straight lines. Suitable line features are the pitch markings and straight edges on buildings 
or advertisement boards. 

If a line in image 1 is represented in homogeneous coordinates as (Li, La, L 3 ) such that the 
point (X, Y) lies on the line if and only if Li X + L* Y + L 3 = 0, and similarly the 
20 corresponding line in image 2 is (L ( \ Ly, I*'), then it is known that 

aL^ + gLa'+dLs' bLi' + hLz'+eLs' 

L, = -■ r --V, U= OH), (TV) 

cL,'+iL 2 '+ fL 3 ' cU' + iW+ fL 3 ' 

The equations for points can be combined with the equations for lines to solve for 
{a,b,c,d,e,f,g,i> simultaneously using linear methods. Alternatively, with a redundant set of 
equations, other error measures can be minimized using known non-linear optimisation 
30 techniques, R. I. Hartley and A. Zisserman. Multiple View Geometry in Computer Vision. 
Cambridge University Press, ISBN: 0521623049, 2000.To suppress noise in the coordinates 
of points and lines, it is advantageous to use more than the minimal number of points and/or 
lines. 



10 : .* : v s ..* J : . j 

Since automatic feature detection and matching is possible once a good initial estimate is 
available as regards a specific set-up of the cameras, it may be possible to avoid any human 
interaction if the apparatus has been used previously, and the cameras 9 relative positions 
and internal parameters (such as focal length) remained similar between its previous use 
and the present. In practice, however, some human interaction to define the lines on the 
pitch and stadium may still desirable to make the procedure more robust to failure of edge 
detection algorithms to find long lines corresponding to pitch markings and other lines in 
the stadium. 
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Rather than using x and y coordinates of point and line features, it is also possible to 
compute the parameters in the tomography from the intensity (bri^tness/colour) values 
using an iterative scheme such as that described in (J.R. Bergen, P. Anandan, K.J. Hanna, 
and R. HingoranL Hierarchical model-based motion estimation, hi Proc. 2nd European 
Conference on Computer Vision, Santa Afdrgharita Ligure, Italy, pages 237-252, 1992.). 

A combination of the line/point features and the intensity feature is within the scope of the 
invention. 



: * *. - - 

*• .* ■' * 

| ■: \ 

;* • > v V \ 
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By setting limitations, i.e. making assumptions on internal parameter of the cameras, e.g. 
assuming that only the focal length is unknown a reduction of the parameter set may be 
accomplished. 

The individual camera images should, preferably, also be corrected for lens distortion. The 
parameters {a,b,c,d,e,f,g^j} are Only the correct description of the inter-camera geometry if 
the idealized pinhole camera model is valid. Many real-world cameras exhibit lens 
distortion (for instance wide-angle lenses usually have barreling distortion) such that lines 
which are straight in the world are imaged as curves. The most significant lens distortion 
may be captured by a single parameter, k, for radial distortion 



30 



Xd=Xo +(Xu-Xo)[1+k(Xu-Xo) 2 ] 05 
Y D =Y 0 +(Yu-Y 0 )[l+K(Y u -Y 0 ) i ] os 



(VI) 
(VII) 
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• •• .... ..- ; 5- 

where (X D ,Y D ) are the actual, distorted coordinates of a pixel, (Xu,Y u ) are the (undistorted) 
coordinates with the corresponding pinhole-lens, and (Xo,Y 0 ) is the centre of distortion, 
assumed to be the centre of the image. 

For each camera, the user may interactively search for a value of K that ensures that lines 
which are straight in the world are imaged as straight lines in the corresponding pinhole- 
lens: given k it is possible, via the equations above, to transform an actual camera image 
into an image which could be produced by a pinhole camera model. These "corrected 
coordinates*' are thereafter used in order to create the wide image. 



The several video sequences are thus, after the distortion correction, temporally 
synchronized. Manual inspection is used to determine corresponding common areas The co- 
ordinates for the corresponding frames are transformed and together they generate the wide 
image video. This transformation is performed such that for every point (X,Y) in the co- 
1 S ordinate system of the wide image video it is decided from which camera of the several 
cameras and from which image co-ordinates (X\Y») the image information is to be 



However, as can be seen in Fig. 6b, illustrating 4 neighboring pixels, in which X'i,Y' x ; are 
20 indicated resulting from the transformation into the individual camera image, the 
coordinates are usually not integers, and therefore the appropriate image value I is 
computed using a weighted interpolation of the image values la, lb, Ic, and Id in the points: 
(X*»Y*X (X*+1,Y*), (X*,Y*+1), (X*+1,Y*+1) wherein (X*,Y*) are integers of (X',Y'). In 
the calibration process for each image point the reference image corresponding coordinates 
25 (X*,Y*) and the weighting factors for the interpolation, which depends on the differences 
between the coordinates (X*,Y*) and (X\Y») is calculated. Thus these need not be 
calculated in the generation of every frame. 

I = (l-dx)(l-dy)I a + (l-dy)dxlb+ (l-dx^ylc + dxdyld, wherein 0£dx£l,0£dy£l,anddx 
= x ' -X* and dy = Y' -Y* . to, .. Ib designates the intensity in each pixel. 



Actually the I-value consists of W 1^, and I bIne so every calculation has in reality to be 
made for 3 colors. This is of course true for any other color space chosen. 
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In order to provide a seamless transition from image 1 to image 2, from camera 1 and 2, 
respectively, the intensities are blended according to 

Iwidc imago 8=5 W1I1+W2I2, LWi=l. (Vm) 

5 This has been suggested by (M. Jethwa,, A. Zisserman and A. W. Fftzgibbon. Real-time 
Panoramic Mosaics and Augmented Reality. Proceedings of the 9th British Machine Vision 
Conference, Southampton 1998,) the weights are a cosine function of the distances di 
between the location (x,y) of that pixel in the wide image and the boundaries of the regions 
in the wide image which each camera covers. 

10 ' ;■''■'/. r . 

wi«0.5(l + cos(di7c/(dt+d 2 ))), W2-I-W1 (IX) 

This is illustrated in Fig. 6c wherein the distances di, and d 2 are indicated in the overlapping 
portion from image A and image B. the pixel, the intensity of which is to be calculated is 
15 indicated at 610. 

This scheme permits a seamless transition from image 1 to image 2 in the wide image; there 
are no visible joins. . 

However, the formula (IX) above assumes that overlapping cameras have the same settings 
20 for exposure and white-balance. If this is not the case (for instance if it is not possible to set 
the cameras' exposure settings manually), and if the difference in settings is too large, there 
is a visible seam in the final wide image. This effect may be largely eliminated by digitally 
adjusting the pre-recorded images from the cameras. A suitable technique is described 
below. 
; -'-'25 

One camera in the pair is considered the reference camera, and its colour settings will 
remain unchanged. For the other camera, a transformation for each colour channel is 
sought which will reduce the seam, thus if the intensity of a channel is Z, we seek a function 
or look-up table f(Z) so that replacing Z «- f<Z) reduces the colour difference between the 
. : 30 images from the two cameras. This can be performed in any colour space (e.g. RGB Red- 
V fc : green-blue or YUV luminance-chrominance channels). In choosing YUV it is usually only 

T-": necessary to modify the brightness (Y) channel if only the exposure settings differ, and the 

' chrominance channels (U and V) can be left unchanged. The ROB is generally used for 

monitors and the YUV is e.g. used in the PAL-system. 



A function f(Z) is found by comparing the histograms of Z, computed in the region of 
overlap in each image. Denoting by hi the histogram of intensities within the region in 
image 1, by hi the histogram of the ^corresponding region in Image 2, and h 2 * thehistogram 
5 of the transformed intensities, we seek a functionf(Z) that minimizes a suitable measure 
similarity between hi and h 2 \ One such suitable measure is the Chi-squared distance 
between the two histograms, 

X 2 = Si(Ri-Si)/(Ri+Si) (X) 

10 

where Rj is the height of bin i in hi and Si is the height of bin i in h£ (Reference: Press et al 
1992), The two histograms are normalized such that the sums of the bins are the same for 
both images, and near-empty bins are not permitted to contribute to the x 2 measure. The x 2 
measure is normalized by the number of bins that contribute to the measure. 

15 . ' : 

A suitable function f (Z) is the parabola Z' ^ f (Z) = a Z 2 + b Z with the constraint a+b=l 
assuming the intensities are normalized to lie in the range 0 to 1. (Usually in digital image 
processing, 8 bits are used for storing each channel, giving a range of 0 to 255. Thus the 
intensity normalization discussed here consists of dividing the intensity by 255.) 
20 Substituting b=l -a, we seek the value a* of a which minimizes the cost function above for a 
in the range [-1,1]. Any standard minimization technique (e.g. exhaustive search, gradient 
descent, or a combination of both) may be applied. Other functions f (Z) can be applied, for 
instance standard gamma correction functions. 

25 Since die exposure settings can change over time, especially with auto-exposure features on 
cameras, the function f (Z) is preferably determined at regular temporal intervals over the 
course of the entire event being recorded or sent live, for instance every 2 seconds, and the 
parameters which describe f (Z) are smoothed over time to prevent sudden 
brightness/colour changes. For instance, using the parabolic function f(Z)= s a Z 2 + bZ,the 

30 parameter a at time t is smoothed temporally using the formula 
at = (l-8)a„i + 8a . . 

and b t is always given by bt = J - a t . 5 is a constant b etween 0 and 1 which determines the 
adaptation rate of a^andbt. 
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The thus generated wide image video sequence may be coded in eg. the Mpeg4 format for 
effective storing on e.g. a digital video disk (DVD). The video sequence may also be 
televised as the generation of the needed at least 30 frames per second (in the US) and at 
least 25 frames per second (in Europe) may easily be accomplished 

The projective transformations between the cameras (also known as inter-image 
"nomographics") encode (i) the relative rotation between the cameras, and (ii) the cameras' 
internal parameters such as focal length, aspect ratio and principal point (Reference: Hartley 
and Zisserman 2000). It is well known mat inter-image homographies can be concatenated 
as follows: Collecting the parameters between cameras 1 and 2 in a 3 x 3 matrix as {[a b 
c],[g h i],[d e f]} denoted Hn , and similarly collecting die parameters describing the 
transformation between cameras 2 and 3 in a matrix H» , camera 1 relates to camera 3 by 
the homography defined by the matrix product H| 2 H23 . This relation generalizes to 
concatenating an arbitrary number of homographies. In this manner all cameras can be 
related to a common reference frame, most simply this common reference flame could be 
one of the central cameras, see Fig. 6d and 6e. 

However, for visually pleasing results, some of the pitch markings should be (a) horizontal 
in the wide image and (b) parallel in the wide image, as in Figure 4b. This adjustment is 
achieved by referring all real cameras to a virtual camera in which these constraints are 
satisfied. 

This virtual camera is related to any previous reference flame by a 3 x 3 homography 
matrix. Suitable techniques for obtaining this virtual view are outlined below. 

1. Two lines on the pitch and/or stadium, which need to be parallel and horizontal in 
the wide image, are identified, for instance by clicking with a mouse on a computer 
display of the wide image. 

Then, EITHER 

2. Perform a self-calibration to obtain the internal parameters and inter-camera 
rotations. Suitable techniques are discussed in (L. de Agapito, E. Hayman, and I. 



is 

Reid. Self-calibration of rotating and zooming cameras. International Journal of 
Computer W5iV>n f 45(2):107-l27,2001.)- 

3. Select a representative set of internal camera parameters as the internal parameters 
of the virtual view. For example, use the internal parameters of one of the central 
cameras. 

4. Compute the rotation angles of the virtual view (relative to a previous reference 
ftame) such that the required lines become parallel and horizontal in this virtual 
view. 

OR - 

2. Select two of the cameras, label their images, Image i and Image j. 

3. Take the eigen-decomposition of their inter-image homography, H i j=WDW" 1 . The 
technique is based on replacing the diagonal matrix of eigenvalues, D, by another 
matrix D' such that the required lines are parallel in the virtual view obtained by 
applying H=WD ; W l to Image i; D' is the diagonal matrix where each eigen-value X 
is replaced by *5. q is found by any standard parameter estimation technique (for 
instance gradient descent) 

2. Apply an image-plane rotation of the virtual cameras, i.e. a rotation about the 
camera's optic axis, such that the required lines become horizontal in its image, and 
hence also in the resulting wide image. 

The two methods are, equivalent if, in the second method, the internal parameters of the two 
selected cameras are the samel In general the internal parameters differ somewhat, but they 
are sufficiently similar to ensure that the second method gives good results in this 
application. 

Due to the wide viewing angle the observer will experience a distortion of depth and width 
relationship in the case when the point of observation deviates from die point which 
corresponds to die focal point of the camera in question. This distortion may partly be 
corrected by modifying the image. 
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The distortion thus may partly be corrected by modifying the image. This modification may 
thus be described, compare above using a non-linear transformation of the co-ordinates for 
the wide image (X, Y) -» (X. ,Y 8 ) - 

5 In viewing football and other sports it is important that the shape of the field is perceived as 
preserved in the image: lines which are straight in reality may not appear too curved in the 
final image. A method which is very well fitted for display of sports is that the X and Y co- 
ordinates are transformed separately. Firstly, an invertable, non-linear transformation 
X S =T|(X, is applied to the X co-ordinates and thereafter an invertable, non-linear 

10 transformation Y 8 =t 2 (Y, ay) is applied to the Y co-ordinates. However, as the 

transformations are independent of each other, T 2 may as well be applied before Ti. 

a x and ay represents a number of parameters defining the transformation. This process 
preserves all horizontal and vertical lines, but not diagonal lines. An example of Ti and T 2 
15 is defined by these inverse transformations. 

X = Ti l (Xs, Ox ; Xo) » Xa + (X.- Xo) [1 + cxx(Xs— Xo) 2 ] 

Y = T2 l (Y s ,a Y ,Y 0 ) = Y 0 +(Y s ^YoHl+a Y (Ys-Yo) 2 ] 

20 . ; ; . : 

Xo is defined as the X co-ordinate for the center line. Yo is chosen as a point between the 
center point of the field and the elongated side of the field farthest away from the cameras. 

ctx and ay are positive parameters which determine how strong the effect of the 
25 transformation is. If Ox is defined by die user first then ay may be determined automatically 
as the values which attains the goal of making the transformed short side of the field as 
straight as possible in the final image. In the same manner ay may be decided by the user 

I and then Ox may be automatically decided on. 

► .* ■ 

■ * 

: 30 Some examples of tins non-linear transformation are given in Fig. 7 a - f for different 
values of Ox and ay. 

Defining the inverse transformations in this manner, implies that for a given point (X s , Y s ) 
in the final image, one point (X, Y) may be found in the uncorrected image. These 
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transformations are combined with the projective transformation to a composite 
transformation. For each value of the final co-ordinates (X,, Y 8 ) it is decide at the time of 
the calibration ftorh which camera and which image co-ordinates are to be used for 
generation of the correctly weighted image value. 

An important aspect of the embodiments according to the invention in order to preserve 
straight horizontal and vertical lines in the wide video image is to use separate non-linear 
transformations of the X and Y co-ordinates and mat the parameters used for a* and a, are 
fitted to each other such that the on-looker experiences a good result 

Thus a number of parameters are gathered at the beginning of the recording session and only 
has to be computed once. Since the cameras do not move during a recording session, the 
spatial calibration parameters remain unchanged and do not need re-computing during the 
sequence. These parameters consist of (i) radial distortion parameters, (ii) projective 
1 5 transformations relating each camera to the reference view, (Hi) parameters describing the 
non-linear transformation applied to the wide image to reduce perspective distortion, (iv) 
parameters selecting a rectangular region of interest in the wide image (this region contains 
the pitch and what parts of the stadium are visible/deemed interesting), and (v) optionally 
an overall scale factor applied to the wide image to reduce storage requirements and enable 
20 the video to be played back on particular hardware. These transformations are concatenated 
to relate which pixel in which camera contributes to which pixel in the wide image: For 
pixel (X,Y) in the final wide image, it is possible to compute the coordinates (X%Y') in 
each camera which correspond to that pixel, as has been explained above. 

25 Since the total transformation does not, in general, yield whole numbers in QC ,Y'), a 

bilinear interpolation scheme is employed such that four pixels in that camera contribute to 
the wide image pixel. The interpolation, coefficients, are denoted ci, 02, C3, C4 where 
ci+c 2 +C3+C4=l . (We refer back to the discussion on p. 9) 

30 If a wide image pixel is visible from two (or more) cameras, the cosine blending scheme 
discussed previously is employed. For n overlapping cameras there are therefore 4n pixels 
which contribute. The coefficients cj from each camera i are multiplied by the weights Wi to 
a give a new coefficient c*y « q Wj , giving a total of 4n coefficients which sum to one. 
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An efficient computer program for generating the wide image video therefore computes the 
coordinates (X\Y*) and the blending coefficients c'y only once per sequence and stores 
them in a table. This table is used in the computer program each time a new set of images 
is obtained from each camera. 

The same table can be used for each channel (RGB or YUV) independently. However since 
the human visual system is 1 ess sensitive to chrominance than to brightness, it is possible to 
use subsampled UV components in a YUV source, for instance subsampling twice in both 
the horizontal and vertical image directions. This requires two tables: one for the Y 
component and another for the UV components. Since video is commonly stored and 
transmitted in subsampled YUV format, it is possible for the entire wide image video 
generation program to use subsampled YUV as its color space. This considerably reduces 
the computation time since the subsampled U and V channels require much less data to be 
processed. 

Below are given some examples of algorithms used in calibrating and video generation. 
The program for calibration is run once per sequence: the calibration parameters found in 
this program may remain constant over the sequence. 

20 In Fig. 8 a flow-sheet describing a process according to the invention for generating a video 
sequence is shown. This is a summary of the important steps in the form of mathematical 
formulas already described above and will not be further discussed here. To be noted is that 
the formula for calculating X' and Y' refers to a point but that the description also relates 
the use of formulas for lines. 

25 . ; .;.'"/ \v . 

A calibration process according to the invention is illustrated in Fig. 9 and comprises the 

■< following steps: 

1. Start of calibration process. 

2. Synchronize the sequences from each camera, which means that at least a video 

30 sequence has to be recorded by all cameras. If the sequences is to be broadcasted live 

all video sequences are of course synchronized from the beginning. 

3; Find the lens distortion parameter(s) for each camera. Correct radial distortion in each 
image produced. 

4. Compute inter-image projective transformations. 
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5. Use the transformations to refer each image to a common reference frame* 

6. Choose a real or virtual reference camera such that certain lines on the pitch and/or 
stadium are essentially horizontal and parallel in die wide image. 

7. Select non-linear distortion parameters to reduce perspective distortion of the wide 
image. 

8. Select a rectangular region of interest within the wide image. This region contains the 
entire pitch and as much of the stadium as is required or visible. 

9. Record all computed values resulting from the calibration process to be used as the 
calibration parameters. 

Among the above mentioned steps step 3 and 7 are optional as they are dependent on the 
cameras used and on the geometry of the view to be recorded or to be televised live. 

The parameters are now there to use. It is of course possible to run the calibration process 
more than one time during a long sequence. 

t' 

A flow chart over an example of a process for the generation of a video sequence according 
to the invention is shown in Fig 10. 

1. Apply the computed and registered calibration parameters. 

For each pixel in the wide image, compute and store parameters describing 

a. Which pixels from which image(s) contributes to this pixel in the wide image. 

b. How much these pixels each contribute to the wide image. 

2. Repeat following steps until the end of the sequence is reached. 

3. Obtain one new image from each camera, either from a live source or a memory 
means. 

4. If required, iipdate thci parameters needed to transform intensities (colours/brightness) 
in one or more cameras to eliminate visible seams. 

5. If necessary, adjust the intensities (colours/brightness) in the images from one or 
more. 

6. Create the current seamless, wide image from the current images from each camera. 

7. Output the wide image to a display or to a (possibly compressed) storage. 

8. End of sequence. Return to step 2 until end of generation of the wide image video 
sequence. ; 



In Fig 11 is shown a device for performing the method according to the invention. With 
reference to Fig. 1 1 there is shown a data processing device 100 for performing the 
methods illustrated in Figures 6-10 comprising a display unit 1 10 for the display of 
information such as text messages or for showing input data- The data processing device 
100 comprises a non volatile memory 120, a microprocessor 130 and a read/write memory 
140. The memory 120 has a first memory portion 121 wherein a computer program is 
stored for controlling fee normal functions of fee data processing device 100. The memory 
120 also has a second memory portion 122, where a program for calibration and recording 
of video sequences and storing of fee resulting wide image video sequence is stored. In 
another embodiment the program for calibration and recording of video sequences and 
storing of fee resulting wide image is stored on a separate non-volatile recording medium 
123. The program may be stored in an executable manner or in a compressed state. 

15 When, in the following, it is described feat fee microprocessor 130 performs a certain 
function this is to be understood feat fee microprocessor performs a certain part of fee 
program which is stored in fee memory 120 or a certain part of fee program which is stored 
on the recording medium 123. 

20 The microprocessor 130 is coupled to fee display unit 1 10 via a data bus 210. A user of fee 
data processing device is provided wife information messages by means of fee program 
stored displayed messages on the display 1 10. A particular message may be displayed in 
response to a certain evenVsuch as for example fee microprocessor having run fee 
calibration part of fee program and fee calibration parameters have been determined, which 

25 may prompt fee microprocessor to display fee message "calibration finished". 

The microprocessor 130 is coupled to the memory 120 by means of a data bus 220 and to 
fee read/write memory 140 by means of a data bus 230. The microprocessor 130 also 
communicates with a data port 300 by means of a data bus 240. 
30 " '! 

The data port is used for input of fee video sequence/s, which in reality may stand for a 
number of input means for input of fee received video signals from several video cameras 
and also as an output means for fee composed wide video sequence. The video sequence 
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may be stored within the device in a separate read and write memory for later retrieval or 
may be forwarded to e.g. a television transmitter. 

In this case the wide image has to be adapted to the format possible for a TV-screen. This 
S may be done such that the wide image is cut by manual input or by an automatic routine 
following a therefore created algorithm. 

In case die video sequence is to be watched concurrently with the recording of the same the 
method and devices are capable of performing the method such that a continuous sequence 
10 may be seen. 

In case the wide image video is to be preserved for die TV-viewer or rather for a viewer of a 
therefore adapted screen the wide image may be cut into parallel images to be sent via 
separate TV-channels and each such image from the concurrently sending TV-channels may 
1 5 thereafter be projected/displayed onto a screen, such that the wide video image sequence 
will again be composed for the viewer to watch. 

The methods described with reference to Figures 8-10 can be performed by the 
microprocessor 130 by means of the microprocessor performing the program stored in the 
20 memory portion 120. In response to an instruction to calibrate, the method described with 
reference to Fig. 9, me microprocessor is set up to follow the steps as described in 
connection with the description of Fig. 9. Likewise in response to an instruction to record 
wide video sequences, the method described with reference to Fig. 10, the microprocessor is 
setup to follow the steps as described in connection with die description of Fig. 10. 

The invention also concerns a customised video player specifically aimed at showing 
widescreen/wide image videos. An example showing the function of this player is shown in 
Fig.l2a-b. .., 

. 30 Fig 12a is esse^ally the same as Fig. 2. Therefore the details of the figure will not be 
described further. 

The field/pitch is indicated as 1210 and the composed wide image area as 1201. Seeing this 
view it can easily be understood that the composed wide image will contain much more 
information than a common video image. 



In Fig. 12b is demonstrated that areas like 1203 and 1202 may be chosen from the entire 
wide video image 1201 . The pitch is as before indicated as 1201. 

5 The user may specify a parameter which controls the scale of the display, allowing the user 
to zoom in if he or she prefers a high-resolution image of part of the pitch rather than a 
lower-resolution display of the entire pitch. This functionality is especially important when 
the video contains more pixels in the horizontal or vertical direction (or both) than the 
display media (e.g. computer monitor, computer projector, or television set) can display. 

io /V- : : : ; , v . 

As the action on the pitch occurs in different locations over time, it is necessary to scroll die 
video in the x and/or y directions. This is accomplished either (i) manually via controls in 
the video player, (ii) by software which invokes an algorithm which automatically identifies 
the interesting region of the pitch, or (iii) using data obtained previously by hand and stored. 

is " / ' 

Although the present invention has been folly described by way of example with reference 
to the accompanying drawings, it is to be understood that various changes and 
modifications will be apparent to those skilled in the art Therefore, unless otherwise such 
changes and modifications depart from the scope of the present invention, they should be 
20 construed as being included therein. 



Claims 

1 . A method for generating a wide image video sequence, said method comprising the 
steps of: 

a. generating a set of calibration parameters related to a device having at least two video 
5 cameras which are arranged in a predetermined relationship to each other, said parameters 

being unique for fee at least two cameras and their current location as related to the object 
being recorded; 

b. recording synchronously video sequences using each of said at least two video cameras, 
and 

10 c. generating a wide image video sequence from each of said synchronously recorded 
video sequences. 

2. A method according to claim 1 in which the synchronously recorded video sequences 
are stored in a memory means. 

is -' 

3. A method according to claim 1 in which the synchronously recorded video sequences 
are used concurrently for generating the wide image video sequence. 

4. A method according to claim 3 in which the wide image video sequence is transmitted 
20 live. V: 

5. A method according to claim 3 in which the wide image video sequence is stored on a 
memory means. 

25 6. A method according to claim 1 in which the generation of calibration parameters comprises 
the following steps: 

a. Start of calibration process; 

b. Synchronize the sequences from each camera, which means that at least a video 
sequence has to be recorded by all cameras; 

30 c. Compute inter-image projective transformations; 

d. Use the transformations to refer each image to a common reference frame; 

e. Choose a real or virtual reference camera such that certain lines on the pitch and/or 
stadium are essentially horizontal and parallel in the wide image; 
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f: Select a rectangular region of interest within the wide image. This region contains 

e.g. the entire pitch and as much of the stadium as is required or visible; and 
g. Record all computed values resulting from the calibration process to be used as the 
calibration parameters. 

7. A method according to claim 6 in which the steps of finding the lens distortion 
parameter(s) for each camera, and correcting radial distortion in each image produced 
are. comprised. . 

8. A method according to claim 6 in which the step of selecting non-linear distortion 
parameters to reduce perapective distortion of the wide image is comprised. 

9. Method according to claim I in which step b is performed manually by identification of 
corresponding features in concurrent video images and the coordinates for these 
corespondning features are input to a computer means. 

10. Memod according to claim 1 in which step bis performed autoniatically by an 
algorithm for identification of corresponding features in concurrent video images and 

15 thecoordhiatesfor ihese corresponding features are mput to a compirtw 

11. Method according to claim 1 which comprises the following steps: 

a. Apply the computed and registered catibration parameters. 

For each pixel in the wide image, compute and store parameters describing 

1. Which pixels from which image(s) contributes to this pixel in the wide image. 

2. How much these pixels each contribute to the wide image; 

b. Repeat until the end of the sequence is reached; 
c Obtain one new image from each camera; 

d. If required, update th e parameters needed to transform intensities 
(colours/brightness) in one or more cameras to eliminate visible seams; 

e. If necessary, adjust the intensities (colours/brightness) in the images from one or 
more cameras; . v 

f. Create the current seamless, wide image from the current images from each 
camera;. ■' 

g. Output the wide image to a display or to a memory means; and 

h. End of sequence. Return to step b until end of generation of the wide image video 
sequence. 

12. Method according to claim 1 1 wherein the new images from each camera are read 
from five sources, each such source comprising a video camera. 
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13, Method according to claim 1 1 wherein the new images from each video camera are 
read from a memory means. 

5 14, In a device having a processor means, which executes instructions stored in at least one 
memory means, a method for generating video sequences comprising the steps of: 

a. generating a set of calibration parameters related to a device having at least two video 
cameras which ate arranged in a predetermined relationship to each other, said parameters 
being unique for the at least two cameras and thekcurreirt to 

10 being recorded; : 

b. recording s^duononsiy video sequences using each of said at least two video cameras, 
and 

c. generating a wide image video sequence by generating from each of said synchronously 
recorded video sequences. 

15 

15. In a device according to claim 14, the method in which the synchronously recoided video 
sequences are stored in a memory means. 

16. In a device according to claim 14, the method in which the synchronously recorded 
20 video sequences are used concurrently for generating the wide image video sequence. 

17. InadeviceaccordmgtocIaiml4^ 
parameters comprises the following steps: 

a. Start of calibration process; 
25 b. Synchronize the sequences from each camera, which means that at least a video 

sequence has to be recorded by all cameras; 

c. Compute inter-image projective transformations; 

d. Use the transformations to refer each image to a common reference frame; 

e. Choose a real or virtual reference view such that certain lines on the pitch and/or 
stadium are essentially horizontal and parallel in the wide image; 

£ Select a rectangular region of mterest within the wide image. This region contains 

the entire pitch and as much of the stadium as is required or visible; and 
g. Record all computed values resulting from the calibration process to be used as the 
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18 In a device according to claim 14, the method in "which the generation of calibration 
parameters the following steps of finding the lens distortion parameters) for each 
camera, and correcting radial distortion in each image produced are comprised. 

, s . * 

19 In a device according to claim 1 4* the method in which the generation of calibration 

parameters the following step of selecting non-linear distortion parameters to reduce 
perspective distortion of the wide image is comprised. 

20. In a device according to claim 14, the method in which step b is performed manually by 
identification of corresponding features in concurrent video images and the coordinates 
for these corresponding features are input to a computer means. 

21. In a device according to claim 14, the method in which step b is performed 
automatically by an algorithm for identification of corresponding features in concurrent 
video images and the coordinates for these corresponding features are input to a 
computer means. 

22. In a device according to claim 9 9 the method which comprises the following steps: 

a. Apply the computed and registered calibration parameter 

For each pixel in the wide image, compute and store parameters describing 

1 . Which pixels from which image(s) contributes to this pixel in the wide image. 

2. How much these pixels each contribute to the wide image; 

b. Repeat until the end of the sequence is reached; 

c. Obtain one new image from each camera; 

d. If required, update the parameters needed to transform intensities 
(colours/brightness) in one or more cameras to eliminate visible seams; 

e. If necessary, adjust the intensities (colours/brightness) in the images from one or 
more cameras; 

f. Create the current seamless, wide image from the current images from each 
camera; 

g. Output the wide image to a display or to a memory means; and 

h. End of sequence. Return to step b until end of generation of the wide image video 
sequence. ^ • 



23. In a device according to claim 22, the method wherein the new images from each 
camera are read from live sources, each such source comprising a video camera. 

24. In a device according to claim 22, the method wherein the new images from each video 
camera are read from a memory means. 

25. A computet readable memory means storing a program which executes the steps of: 

a. generating a set of caUhiation parameters related to a device having at least two video 
cameras which are arranged in a predetermined relationship to each other, said parameters 
being unique for the at least two cameras and their current location as related to the object 
being recorded; 

b. recording synchronously video sequences using each of said at least two video cameras, 

and ': ., "• 

c. generating a wide image video sequence by generating from each of said synchronously 
recorded video sequences. .. 

26. A memory means storing a program according to claim 17, in which the synchronously 
. recorded video sequences are stored in a memory means. 

27. A memory means storing a program according to claim 17, in which the synchronously 
recorded video sequences are* used concurrently for generating the wide image video 
sequence. 

28. A memory means storing a program according to claim 17, in which the generation of 
calibration parameters comprises the following steps: 

a. Start of calibration process; 

b. Synchronize the sequences from each camera, which means that at least a video 
sequence has to be recorded by all cameras; 

c. Compute inter-image projective transformations; 

d. Use me transformations to refer each image to a common reference frame; 

e. Choose a real or virtual reference view such that certain lines on the pitch and/or 
stadium are essentially horizontal and parallel in the wide image; 



f. Select a rectangular region of interest within the wide image. This region contains 
the entire pitch and as much of the stadium as is required or visible; and 

g. Record all computed values resulting from the calibration process to be used as the 
calibration parameters. 

29. A memory means storing a program according to claim 28, in which the steps of 
finding the lens distortion parameters) for each camera, and correcting radial 
distortion in each image produced are comprised. 



30. A memory means storing a program according to claim 28, the step of selecting 
linear distortion parameters to reduce perspective distortion of the wide image is 



non- 



31. A memory means storing a program according to claim 28, in which step b is 

performed manually by identification of corresponding features in concurrent video 
images and the coordinates for these corresponding features are input to a computer 



means. 



32. A memory means storing a program according to claim 28, in which step b is 
performed automatically by and algorithm for identification of corresponding features i 
concurrent video images and the coordinates for these corresponding features are input 
to a computer means. 

33. A memory means storing a program according to claim 28, which comprises the 



a; Apply the computed and registered calibration parameters. 

For each pixel in the wide image, compute and store parameters describing 

1. Which pixels from which image(s) contributes to this pixel in the wide image. 

2. How much these pixels each contribute to the wide image; 

b. Repeat until thie end of the sequence is reached; 

c. Obtain one new image from each camera; 

d. If required, update the parameters needed to transform intensities 
(colours/brightness) in one or more cameras to eUminate visible seams; 



e. If necessary, adjust the intensities (colours/brightness) in the images from one or 
more cameras; 

f. Create the current seamless, wide image from the current images from each 

' ■ ''-V' : ' • ' 

camera; - ; '. 
S g / Outouttoevwdei^ 

h. End of sequence. Return to step b until end of generation of the wide image video 

sequence. 

34. A memory means according to claim 28, wherein die new images from each camera 
10 are read from live sources, each such source comprising a video camera. 

35. A memory means storing a program according to claim 28, wherein the new images 
from each video camera are read from a memory means. 

15 36. A video recording apparatus comprising: 

a microprocessor(130), a memory means (120) for storing program for generating a set of 
calibration parameters related to a device having at least two video cameras which are 
arranged in a predetermined relationship to each other, said parameters being unique for 
ihe at least two cameras and their current location as related to me object being 

20 recorded; 

said memory means (120) also storing program for recording of wide image video sequences; 
read and write memory means (1 40) for storing data relating to recorded video sequences 

from at least two video cameras; 
input means (300) for input of manual input of parameters, input of recorded video sequences, 
25 and output means (300) for output of a wide image video sequence. 
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The invention relates to a 
a microprocessor(l 30) s a memory means 

calibration parameters related to a device having ; at least tWo video cameras which are 
arranged in a predetermined relationship to each other, sirid parameters being unique for the at 
least two cameras i 
said memory means I 




fiom at least two video cameras; 

input means (300) for ftput itf manual input of pa^^ra, input of recorded video sequences, 
output means (300) for output of a wide image video sequence. 
The invention also relates to a method for generating a 1 
method comprising tne steps pi generating a s 

other, j 

related to the obje 

J; " '."/.•* (>,;"."♦•, ,' •■. . "' r //- '>r • 

said at least two video cameras, and generating a wide image video sequence ftom each of 

said synchronously recorded video sequences. Y^^f^t;] 
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Fig. 5 
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Fig. 6 b 
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individual images recorded 
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Fig. 6c 
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Fig. 8 
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to cov» approximately even areas giving an 
angle view of approximately 120 --10 . 
degrees. I '-: . ; '?>: C \ 



• r I Chose a reference camera. 



identify manually a predetermined number of 
common points in an overlap region of a picture 



frorii the reference camera and the n* camera. 
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Calculate \ff^Sfi 9 where n is the n* camera, using 



Repeat for all cameras up to the y^cfim Form a 
matrix for the corresponding X\ and Y % n aad X and Y to be 



Determine the integers X*,Y* coiresponcling to X n and Y % n . 
Add to the matrix. Use the integers for calculating a 
weighted picture value to be applied to corresponding X,Y 
in the wide pictuire. 



Apply separate transformations to the X and Y 
coordinated, providing X+ and Y+ iri order to transform 
the final picture such that straight linfes appear as straight 
as possible, : : \i 



• V;.-*. ' 

. *■ *..•*: • • 
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Start of calibration process 
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Synchronize the sequences 
from each camera . 
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If the sequence is sent live the there is 
no need for manual synchronization 



"j Find the lens distortion parameter(s) 
' ; for each camera. Correct tadial 
distortion in each image (if nectary) 



Compute inter-image projective 
, transformations i:, v 



1 



; (Jse the transformations to refer each 
. ' image to a common reference frame 



Choose a real or virtual reference camera 
such that certain lines on the pitch and/or 
stadium are essentially horizontal and 
-^'pm^ in the wide image: 



Select non-linear distortion 



distortion of the wide image ; if 
: -.4'V' - necessary' 



Select a rectangular region of , 
interest within the wide image. : 

This region contains the entire 
C' pitch and as much of the ' ; v 
stadium as is required oi visible 
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Record all computed values resulting 
from the calibration process to be used as 
thfe calibration parameters v 




^\ either from a live soi^e br a memory 



transform intensities (coloiifs/bri^itness) in one 
or more cameras to eliminate visible seams 
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% If necessary, adjust the intensities 
; (colours/brightness) in &fe i^ge$; 
'J;.-: "from one or more ^aai&^ : ii . 



Create the current seamless, wide image 
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Output the wide image to a display or to 
5 a (possibly compressed) storage 
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BEST AVAILABLE IMAGES 

Defective images within this document are accurate representations of the original 
documents submitted by the applicant. 

Defects in the images include but are not limited to the items checked: 

□ BLACK BORDERS 

□ IMAGE CUT OFF AT TOP, BOTTOM OR SIDES 
^FADED TEXT OR DRAWING 
JZi^LURRED OR ILLEGIBLE TEXT OR DRAWING 

□ SKEWED/SLANTED IMAGES 

□ COLOR OR BLACK AND WHITE PHOTOGRAPHS 

□ GRAY SCALE DOCUMENTS 

[j LINES OR MARKS ON ORIGINAL DOCUMENT 

□ REFERENCE(S) OR EXHIBIT(S) SUBMITTED ARE POOR QUALITY 

□ OTHER: , 

IMAGES ARE BEST AVAILABLE COPY. 
As rescanning these documents will not correct the image 
problems checked, please do not report these problems to 
the IFW Image Problem Mailbox. 



